The use of standardized testing has increased significantly in recent years. Indeed, standardized tests are now an essential part of almost every admission or qualification procedure. A recent trend in standardized testing emphasizes a move beyond traditional multiple choice tests in favor of tests that require open-ended responses such as essay responses. These open-ended responses are often referred to as constructed responses (CRs). CRs are not limited to written text, but may include graphics, videotaped performances, audio responses, as well as other forms of responses. In order to improve the efficiency of scoring large scale standardized tests, both those offered at periodic administrations as well as those offered essentially on a daily basis, computer systems have been developed to automatically score multiple-choice responses and other simple response types. While some automatic scoring systems have been designed to score particular types of CRs (see e.g., U.S. patent application Ser. No. 08/794,498, entitled Automatic Scoring for Categorized Figural Responses, assigned to the same assignee hereunder), the evaluation of CRs is particularly well-suited to human raters. For this reason, certain computer scoring systems have been developed to facilitate and automate the electronic transmission of CRs to human raters for evaluation and scoring. However, these conventional CR transmission systems currently have many disadvantages.
Generally, essay evaluation systems operate in the context of a particular test administration. The systems are used to collect, distribute, and grade actual test responses for a particular test. There is not presently available a system which provides the capability of practicing test taking skills, demonstrating content mastery, and receiving constructive feedback from qualified faculty who have scored actual test responses. Such a system would have tremendous benefits for those persons interested in improving essay writing scores and those preparing for tests. Thus, there is a need for a system that allows persons to improve testing skills by formulating responses to actual test questions in a practice setting and having their responses evaluated by actual test evaluators.
Current evaluation systems also lack an adequate means for scheduling readers/graders. For a system which can accept large numbers of constructed responses, a large number of readers are required to evaluate the constructed responses. Such a system should track when readers are scheduled to be on the system and if scheduled readers do not log on or grade a sufficient number of essays, to notify backup readers.
Some prior systems permanently associate multiple essays or constructed responses with a particular evaluator or reader. The constructed responses remain associated with the reader regardless of whether the reader is logged onto the system. In such systems, when the reader is not logged-in, the constructed responses sit idle waiting for the reader to evaluate the response. A constructed response may wait in a reader's queue for several days before the reader evaluates the response. Under such a system it is not possible to control when a constructed response may be evaluated. Thus, there is a need in the art for a system for evaluating constructed responses that provides automated work load leveling of queued essay responses so that essay responses are evaluated and returned to the submitter or student within a predetermined time.
A further shortcoming of prior systems is the inability to prioritize constructed responses for grading. There is not presently available a means to expedite grading of particular constructed responses. Further, current systems do not account for such prioritization markings in the routing of constructed response to readers so that the desired deadlines are satisfied. Thus, there is a need in the art for a system which allows users to prioritize their essays and which automatically routes essays to evaluators so that the desired dates for evaluation are met.
Another shortcoming in the art is the inability to automatically determine reader compensation based on work performed. Typically, in prior systems, a reader or evaluator is paid a flat fee for participation in a particular scoring project. In an optimal system, a reader's compensation is based upon the reader's work product, i.e., the number of constructed responses that were graded. Further, in an optimal system, the compensation might vary between questions. Thus, there is a need for a system which provides a means for calculating compensation based on the work actually performed by the reader.
An additional shortcoming in present systems is the inability to monitor reader evaluations so as to ensure consistency and quality in scoring. Without such a mechanism, scoring is random and greatly diminishes the usefulness of the system to a student who needs consistent scoring in order to measure improvement. Thus, there is a need for a system whereby reader evaluations are monitored so as to insure quality and consistency in grading.
Still a further shortcoming in present systems is the inadequate feedback that is provided to a student or user who has submitted an essay. Specifically, present systems lack the capability to provide consistent essay feedback from multiple readers and fail to provide students with samples of previously graded essay responses. Typically, in present grading systems, a single numerical score is assigned to a constructed response. In such systems, the user receives no feedback other than an overall score. Further, in those systems that provide additional feedback other than an overall score, the responses are not consistent and non-standardized between readers; generally, the feedback comprises free-hand comments by the reader. The lack of consistent feedback diminishes significantly the benefit of the scoring system to the user or test-taker. Additionally, present systems do not provide the user with samples of scored essays for the same question. Thus, there is a need in the art for a scoring system that provides consistent feedback and sample essay responses.